Representing Semantic Relationships in Ancient IE Languages

Word Embeddings: Overview

Second approach is one more in line with what is done in present-day NLP – identifying semantic relationships through word embeddings.
To do so, we must:
- Process a corpus for tokens and then lemmas;
- Analyze the environments in which these lemmas occur;
- Take this information to construct a semantic “hyperspace”

Plots like these can be created with generated vectors!

If you were to plot out every vector generated from one of these models, you would have a hyperspace or a semantic space, with the dimensions of each word vector essentially acting as coordinates.
The closer two words lie to each other in this space, the closer in semantic value they are.
It’s possible to adjust how many dimensions you generate for each vector, but in general the more the better.
- More fancy math allows us to ‘simplify’ these vectors into 2 or 3 for easy viewing, as seen before.

Using a word embeddings model such as word2vec or fastText, run the tokenized and lemmatized text through it to generate word vectors.
These vectors will be based on the words’ positions within the text.

To get a numerical representation of the similarity between two vectors, we use their cosine similarity (cosine of the angle between the vectors).
Similarity scores range from -1 to 1, with -1 indicating opposite vectors and 1 indicating proportional vectors.

Fancy formula from Wikipedia

We take two, similar properties in two related languages, which allow us to approximate an earlier state in a source language

It is in this way that we propose that the use of word embedding models created by descendant languages, to approximate an earlier state in the source language

As you can imagine, this stuff is complicated, which is why we won’t go into much detail about the specific methods;
- see Github for four-page paper, code, data, etc.
If there are any questions that we can’t answer, we’ll forward them to our main collaborator, Anton, who will be happy to do so

Vectors generated for hyperspaces take on arbitrary values when training models
- So ‘dog’ could = (0, 0) or (-6, 100) – these values change every time you run the model
For this reason, we must align models (Dev et al., 2021):
- Identify substructures across language models that remain fixed
- Use pre-aligned word embedding models (following Joulin et al., 2018)

So we take pre-aligned hyperspaces, and reconstruct an earlier, source hyperspace based on the hyperspaces provided
But how can we trust this methodology?
- Obviously can’t verify the hyperspace of PIE through analysis of PIE texts!

We use existing aligned models of Spanish & French (Joulin et al., 2018) as a source of vector and word information
- Aligned word vectors taken from Facebook AI Research’s fastText (Bojanowski et al., 2017)
Words filtered out:
- If there is no corresponding word in the other languages
- Non-vocabulary, including words with non-language characters

Models trained on French & Spanish Wikipedia articles, include both vocabulary and non-words/words containing non-language characters, the latter of which were removed (7.3%, 6.6%, respectively)
Remaining words lemmatized, furthering reducing vocabulary by roughly 10%

To relate words together and find common words:
- Words are translated into each other’s respective languages, using Google Translate, with a Python translation library deep-translator
- The same is done with Latin, using the Latin corpus (from CLTK [the Tesserae Project]), which is lemmatized
Any word that cannot be lemmatized in Latin (such as Greek words) is removed from the corpus

If there isn’t a 1:1 correspondence between the Romance language & Latin, identify the centroid of lemma’s vector: language-word center
Identify the centroid of both lwcs -> inter-language-word center
Identify the closest vectors to the ilwc using cosine-distance
Take the average of these two vectors to arrive at the approximate *Latin word vector

To identify the effectiveness of the method, recall that we want to compare the *Latin with the Latin
This is our “normal” model

The analogy task (Mikolov et al. 2013) is considered standard when evaluating word embedding models: London is to England as Paris is France
But it doesn’t work for languages with small corpora (LRLs), especially ones that aren’t modern
We follow Stringham & Izbicki 2020 in using the OddOneOut task, which has been demonstrated to be more accurate in these situations
OddOneOut task demonstrated to work for corpora as small as 1800 tokens (Old Gujarati)

Our results indicate mixed success: we can create a hyperspace from the descendant languages that performs reasonably well on Latin tests with OddOneOut
Preliminary tests show that the smaller the corpus, the better the model works, as compared to the normal one
This is not exactly a bad thing, as this is the case for many languages in our discipline!
- If a language has a large corpus (such as Ancient Greek), we can reduce it as we’ve done here for Latin

Unclear what is the “magic number” for the size of corpus to arrive at most accurate representation (compared to the normal model)
Unclear exactly why performance of descendant models tends to decrease as the corpus size increases
- Anton has ideas, all quite technical – see the draft for further

Use of Google translate suboptimal, may result in translation errors; should use either bilingual dictionaries or LLMs (think GPT) for more accurate translations
LLMs > Word Embeddings
1. Vectors: polysemy (e.g., ‘bank’)
2. Vectors: context
3. Vectors: precision (300 vs. 175B parameters)